Apparatus and method of building an electronic database for resolution synthesis

ABSTRACT

An electronic database for image interpolation is generated by a computer. The computer generates a low-resolution image from a training image, a plurality of representative vectors from the low-resolution image, and a plurality of interpolation filters corresponding to each of the representative vectors. The interpolation filters and the representative vectors are generated off-line and can be used to perform image interpolation on an image other than the training image. The database can be stored in a device such as computer or a printer.

[0001] This is a continuation-in-part of Ser. No. 08/837,619 filed onApr. 21, 1997.

BACKGROUND OF THE INVENTION

[0002] The invention relates to digital imaging. More specifically, theinvention relates to interpolation of higher resolution images fromlower-resolution images.

[0003] Image interpolation is performed routinely by printers andcomputers. In one instance, a printer might receive source image datahaving a fixed resolution of 75 dots per inch (dpi), yet the printer iscommanded to print at a higher resolution such as 300 dpi. In such aninstance, the printer performs interpolation on the source image data.

[0004] In another instance, a computer might perform interpolation inorder to display an image compressed according to a lossy algorithm suchas JPEG. Once the image is displayed, the computer might receive a usercommand to zoom in on a portion of the image to magnify a particulardetail. In response to the user command, the computer would performinterpolation on the pixels in that portion.

[0005] Simple linear interpolation lacks the precision necessary toreproduce an image with quality at a higher resolution. Edges in theinterpolated image have poor quality. Shading problems and jaggedtransitions can be viewed when an interpolated image is displayed orprinted. The shading problems and jagged transitions become even worsewhen a region of the interpolated image is magnified.

[0006] Low order B-spline methods such as pixel replication and bilinearinterpolation are satisfactory in terms of interpolating smoothtextures, and they are easy to implement in terms of processing powerand memory requirements. However, pixel replication tends to producesharp edges that are not straight, and bilinear interpolation tend toproduce images that include artifacts and relatively blurry lines.

[0007] Higher-order B-spline interpolation methods such as cubicB-spline interpolation tend to provide smooth, continuous images.However, smoothness and continuity come at a cost: increasedcomputational power. Higher-order B-spline interpolation methods arecomputationally intensive. They also give ringing effects. Additionally,B-spline methods in general are linear methods and, therefore, arelimited in terms of quality of the interpolation.

[0008] Other interpolation methods include edge-directed methods,fractal interpolation and methods that employ stochastic models. For anexample of a method that employs a stochastic model, see Schultz et al.,“A Bayesian approach to image expansion for improved definition,” IEEETransactions on Image Processing, vol. 3, no. 3, pp. 233-242 (May,1994). The method disclosed therein is computationally intensive.

[0009] There is a need for an image interpolation method that produceshigh quality images, yet is easy to implement in terms of processingpower and memory requirements.

SUMMARY OF THE INVENTION

[0010] The invention can be regarded as a method of generating anelectronic database of interpolation parameters that can be used toproduce high quality images. The parameters can be computed from one ormore training files. A low-resolution file is computed from eachtraining file. A plurality of representative vectors from eachlow-resolution file are then computed. Next, a set of interpolationfilters are computed for each of the representative vectors. Theinterpolation filters and the representative vectors can be used toperform interpolation on a file other than the training file.

[0011] The method of generating the electronic database can be performedoff-line. Thus, by the time the parameters in the database are used forinterpolating an image, the bulk of the computational activity hasalready been performed. Consequently, interpolation time and memoryrequirements are reduced.

[0012] The training file could be an image file. The electronic databasecould be stored in a device such as computer or a printer.

[0013] Other aspects and advantages of the present invention will becomeapparent from the following detailed description, taken in conjunctionwith the accompanying drawings, illustrating by way of example theprinciples of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014]FIG. 1 is a block diagram of a system according to the presentinvention, the system including a printer and first and secondcomputers;

[0015]FIGS. 2a, 2 b and 2 c are flowcharts of different ways in whichthe first computer sends an image to the printer and the printer printsan interpolated image;

[0016]FIG. 3 is an illustration of an image interpolation methodaccording to the present invention;

[0017]FIG. 4 illustrates the generation of a cluster vector, which formsa part of the image interpolation method according to the presentinvention;

[0018]FIG. 5 is an illustration of a non-linear filtering operation,which forms a part of the image interpolation method according to thepresent invention;

[0019]FIG. 6 is a method of generating a database of parameters for thenon-linear filtering operation; and

[0020]FIG. 7 is a more generalized method of generating the database.

DETAILED DESCRIPTION OF THE INVENTION

[0021] As shown in the drawings for purposes of illustration, thepresent invention is embodied in a method of generating an electronicdatabase of parameters used for interpolating higher resolution imagesfrom lower resolution images. An interpolation method uses theparameters to produce images of high quality, yet the interpolationmethod is easy to implement in terms of processing power and memoryrequirements. The electronic database can be used advantageously by asystem including a computer and a printer. However, as will be discussedbelow, the invention is not limited to generating a database for imageinterpolation by the computer or printer. More generally, the inventioncan be applied to the restoration or enhancement of speech, stillimages, video and other multidimensional data.

[0022]FIG. 1 shows a system 10 including a first computer 12 and aprinter 14 that communicate over a cable 16. A resolution synthesisdatabase 18 is stored in either computer memory 20 (e.g., a hard drive,an EEPROM) or in printer memory 22 (e.g., an EEPROM). The database 18provides resolution synthesis parameters that are used for interpolatinghigher resolution images from lower resolution images. Resolutionsynthesis (i.e., interpolation using the resolution synthesis database18) can be performed by a host processor 24 in the first computer 12 orby an embedded processor 26 in the printer 14.

[0023] The database 18 includes a first group of parameters forresolution synthesis at a first interpolation factor, a second group ofparameters for resolution synthesis at a second interpolation factor, athird group for resolution synthesis at a third interpolation factor,and so on. Any positive integer could provide a suitable interpolationfactor. If, for example, the image has a resolution of 75 dpi, but itwill be printed at a resolution of 300 dpi, a group of resolutionsynthesis parameters corresponding to an interpolation factor of fourwill be accessed from the database 18.

[0024] The database 18 can be generated off-line by the first computer12 or another (second) computer 2. The second computer 2 includes aprocessor 4 and computer memory 6. A program 8 is stored in the memory 6of the second computer 2. The program 8 includes a plurality ofexecutable instructions that, when executed, instruct the processor 4 ofthe second computer 2 to generate the database 18. A method ofgenerating the database 18 will be described below in connection withFIGS. 6 and 7. Once computed, the database 18 is transferred from thesecond computer 2 to the computer memory 20 of the first computer 12(via a CD ROM, for example) or to the printer memory 22 of the printer14. The method of generating the database 18 is performed off-line.Since the bulk of the computational activity is done prior to the actualinterpolation, interpolation time and memory requirements of the firstcomputer 12 and the printer 14 are reduced.

[0025] Additional reference is now made to FIGS. 2a, 2 b and 2 c, whichillustrate three different examples in which the first computer 12 andthe printer 14 use the database 18 to perform resolution synthesis. Inthe first example, as shown in FIG. 2a, the database 18 is stored incomputer memory 20. To print an image in a source image file, a printerdriver 19 instructs the first computer 12 to access the database 18 fromits memory 20 (block 200), perform resolution synthesis on the sourceimage file to compute an interpolated image (block 202), and send a file(block 204) including the interpolated image to the printer 14 forprinting.

[0026] In the second example, as shown in FIG. 2b, the database 18 isalso stored in computer memory 20, but the first computer 12 sends thesource image file and a file including the database 18 to the printer 14(block 210). Under instruction of a program 21 stored in the printermemory 22, the embedded processor 26 of the printer 14 performsresolution synthesis on the source image file (bock 212) and prints outthe interpolated image.

[0027] In the third example, as shown in FIG. 2c, the database 18 isstored in the printer memory 22. The computer 12 sends only the sourceimage file to the printer 14 (block 220), which performs resolutionsynthesis on the source image file (block 222) and prints out theinterpolated image.

[0028]FIG. 3 illustrates a method of performing the resolution synthesison a pixel L4 of an input image 30. An interpolation factor of 2 will beused by way of example. A small observation window 32 is located aboutthe pixel L4 to be interpolated (the “sampled pixel”) and encompassesneighboring pixels of the sampled pixel L4. The observation window 32provides sufficient information for the sampled pixel L4 to beclassified and interpolated The observation window 32 could be a 3×3window, a 3×5 window, a 5×5 window, or larger. By way of example,however, the observation window 32 will hereinafter be described as a3×3 window centered about the sampled pixel L4.

[0029] An observation vector L and a cluster vector y are extracted fromthe observation window 32. The observation vector L is extracted bystacking pixel values (e.g., intensity values of the luminancecomponent) of the pixels in the observation window 32. Thus, nine pixelvalues would be stacked into the observation vector L extracted from a3×3 window. Such an observation vector L would be represented as

L={L0, L1, L2, L3, L4, L5, L6, L7, L8}^(t)  (1)

[0030] where t is a vector transpose.

[0031] The cluster vector y is extracted (by block 34) using a nonlinearprocess. The cluster vector y contains information about a general typeof image behavior (e.g., an edge, a smooth surface) in the observationwindow 32. To begin cluster vector extraction for the 3×3 window shownin FIG. 3, the eight nearest neighbors of the sampled pixel L4 arestacked into a first vector y1, where

y1={L0, L1, L2, L3, L5, L6, L7, L8}^(t).  (2)

[0032] A nominal value such as the value of the sampled pixel L4 issubtracted from each value in the first vector y1 to yield a secondvector y2:

y2={L0−L4, L1−L4, L2−L4, L3−L4, L5−L4, L6−L4, L7−L4, L8−L4}^(t).  (3)

[0033] Thus, the second vector y2 indicates differences between thesampled pixel L4 and its neighboring pixels. The cluster vector y iscomputed by scaling the second vector y2. Scaling can be performed byperforming a projection operation f on the second vector y2. Anon-linear scaling operation might be performed as follows:$\begin{matrix}{{{y2}} = \sqrt{\sum\limits_{\underset{\underset{\underset{i \neq 4}{\quad}}{\quad}}{i = 0}}^{i = 8}\left( {{L\quad i} - {L4}} \right)^{2}}} & (4) \\{y = {{f({y2})} = \left\{ \begin{matrix}{{y2}{{y2}}^{p - 1}} & {{i\quad f\quad {y2}} \neq 0} \\0 & {e\quad l\quad s\quad e}\end{matrix} \right.}} & (5)\end{matrix}$

[0034] where p is any scalar between zero and one. For example, thescalar p can equal 0.25. The projection function ultimately affects howwell the edges and details are rendered in the interpolated image. Suchgeneration of the cluster vector y is illustrated in FIG. 4.

[0035] A non-linear filtering operation is performed (by block 36) onthe 9-dimensional observation vector L and the 8-dimensional clustervector y. Parameters used to perform the non-linear filtering operationare accessed from the database 18. An output of each non-linearfiltering operation produces a 2×2 window 38 of interpolated pixels H0,H1, H2, H3.

[0036] Thus, interpolation of a single pixel L4 in the input image 30has been described. To interpolate the other pixels in the input image30, the observation window 32 is moved around from pixel to pixel in theinput image 30. For each non-border pixel L4 in the input image 30, fourinterpolated pixels H0, H1, H2, H3 are produced by the non-linearfiltering. Border pixels can be interpolated by substituting the closestknown pixels for all unknown pixels. A border pixel might be a pixelthat cannot be centered within the observation window 32. After all ofthe pixels in the input image 30 have been interpolated, there is aninterpolated image 40 having a resolution twice that of the input image30.

[0037] The interpolation method can be applied to monochrome images andcolor images. When a color image is interpolated, cluster vectors from aluminance component of the color image are extracted, and the clustervectors are used to perform non-linear filtering in each of theconstituent Red-Green-Blue planes of the color image. Thus, the samedatabase parameters are used to perform non-linear filtering on the Redplane as are used to perform non-linear filtering on the Green and Blueplanes.

[0038]FIG. 5 shows the non-linear filtering operation in detail. Inputsinclude the resolution synthesis parameters stored in the database 18.The parameters include a number M of representative vectors RV, a numberM of class weightings CW, a variance VAR and a set of interpolationfilter coefficients a and b. The representative vectors RV, the relativeclass weights CW and the variance VAR will collectively be referred tohereinafter as a “classifier.” Each representative vector RV representsa different image behavior. Examples of image behavior includehorizontal edges, vertical edges, angled edges, smooth textures, etc.There might be between 50 and 100 representative vectors RV stored inthe database 18. Thus, there might be between 50 and 100 representativevectors RV that can be used to synthesize the interpolated image 40 fromthe input image 30. The class weights CW correspond to how often thedifferent representative vectors RV or behaviors occur. They convey therelative frequencies of the representative vectors RV. Thus, the classweight for flat pixels would typically be quite large relative to theclass weight for vertical edges. The variance VAR is a positive numberrepresenting the level of variation within classes associated with therepresentative vectors.

[0039] The pixels in the input image 30 are sampled one at a time. Whena pixel is sampled, its observation vector L and its cluster vector yare also provided as inputs to the non-linear filtering operation.

[0040] The cluster vector y is classified to determine the type of imagebehavior within the observation window 32 (block 42). That is, thecluster vector y is classified to determine whether the sampled pixel ispart of a horizontal edge, a smooth texture, etc. A classificationparameter p(j|y) is computed for j=0 to M−1 to indicate therepresentative vector RV or vectors RV that best represent the clustervector of the sampled pixel. The classification parameter p(j|y) for acluster vector y can be computed as follows: $\begin{matrix}{{{p\left( j \middle| y \right)} = \frac{C\quad W_{j}{\exp \left( \frac{- {{y - {R\quad V_{j}}}}^{2}}{2{VAR}} \right)}}{\sum\limits_{d = 0}^{M - 1}\left( {C\quad W_{d}{\exp \left( \frac{- {{y - {R\quad V_{d}}}}^{2}}{2{VAR}} \right)}} \right)}}{w\quad h\quad e\quad r\quad e}} & (6) \\{{{y - {R\quad V_{j}}}}^{2} = {\sum\limits_{m = 0}^{7}\left( {{y(m)} - {R\quad {V_{j}(m)}}} \right)^{2}}} & (7)\end{matrix}$

[0041] In equation (7), y(m) is the m^(th) element in the cluster vectory, and RV_(j)(m) is the m^(th) element in the j^(th) representativevector RV.

[0042] Equations (6) and (7) are derived under the assumption that aparticular behavior might encompass a population of cluster vectors,with certain cluster vectors being more likely to fall under thatbehavior than others. Take an example in which pixels in the upper halfof the observation window 32 are at full intensity while pixels in thelower half of the observation window 32 are at zero intensity. A clustervector extracted from such a window 32 would indicate with highprobability that the sampled pixel in the observation window 32 is partof an edge. If, however, the pixels in the lower half of the observationwindow 32 are half-scale, the probability is not as high.

[0043] Moreover, the equations (6) and (7) are derived under theassumption that the cluster vectors y are distributed as a multivariateGaussian mixture. Taken as a group, the distributions of the variousbehaviors form a Gaussian mixture model. The distributions overlap.Consequently, a cluster vector (such as the cluster vector for anobservation window having the upper half of its pixels at full scale andthe lower half of its pixels at half-scale) might indicate a behaviorthat falls under multiple distributions. Thus, classification of acluster vector y according to equations (6) and (7) is equivalent tocomputing probabilities of class membership in the Gaussian mixturemodel. The classification parameter p(j|y) for a cluster vector yindicates a series of weightings or probabilities corresponding to thedifferent behaviors and, therefore, the different representative vectorsRV.

[0044] Classifying a cluster vector y reveals the set or sets ofinterpolation filters 44 that should be used to interpolate the sampledpixel. Each set of interpolation filters 44 corresponds to arepresentative vector RV. After a cluster vector y has been classified,the corresponding observation vector L is filtered by the coefficientsa(j,k), b(j,k) of the selected interpolation filter or filters 44, andthe output of each filter 44 is multiplied (by blocks 46) by theprobability that the observation vector L is represented by therepresentative vectors RV corresponding to that interpolation filter.Products of the multiplication are summed together (block 48). Thus, aninterpolated pixel H(k) is computed as follows: $\begin{matrix}{{H(k)} = {\sum\limits_{j = 0}^{M - 1}{\left( {{{a\left( {j,k} \right)}L} + {b\left( {j,k} \right)}} \right){p\left( j \middle| y \right)}}}} & (8)\end{matrix}$

[0045] for k=0, 1, 2, and 3, where H(k) is the k^(th) interpolated pixelin a vector h, and a(j,k) and b(j,k) are scalars representing the filtercoefficients for the j^(th) class and the k^(th) interpolated pixel.

[0046] A computation could be performed for each interpolation filter44, and outputs of all of the interpolation filters 44 could bemultiplied by the classification parameter p(j|y). However, if acomputation is to be performed for each interpolation filter 44,computational time might be excessive. Moreover, interpolating withrepresentative vectors RV that are different from the image data mightresult in a waste of computational resources. Therefore, a differentnon-linear filtering operation might use the outputs of only one or twoof the interpolation filters 44. This different operation would be morecomputationally efficient if many interpolation filters 44 wereinvolved.

[0047] The interpolation method described in connection with FIGS. 3, 4and 5 is also described in U.S. Ser. No. 08/837,619 filed Apr. 21, 1997and entitled “Computerized Method for Improving Data Resolution.” U.S.Ser. No. 08/837,619, which is assigned to the assignee of the presentinvention, is incorporated herein by reference.

[0048]FIG. 6 shows a training method of generating the resolutionsynthesis parameters for the electronic database 18. The parameters arecomputed from at least one, but preferably more than one training image.Training images can include photo-quality color images, 600 dpi blackand white images, etc. A collection of digital images are commonlyavailable on photo CD and might provide a possible source for thetraining images. To simplify the description of the training method, thetraining method will hereinafter be described in connection with only asingle training image.

[0049] The general premise of the training method is to generate acorrupted (e.g., low resolution) image from the training image and thencompute a set of resolution synthesis parameters that can be used topredict or estimate the uncorrupted training image from the corruptedtraining image.

[0050] The training image is converted to a monochrome image, ifnecessary (block 100). For example, if the training image is a colorimage, it can be converted to a representation having a single plane. Inthe alternative, the luminance component of the color image can beextracted, whereby values of the pixels in the observation vector Lwould represent intensities of pixels in the luminance component.

[0051] A scaled-down or low resolution rendering of the monochrome imageis computed (step 102). The interpolation factor dictates the level ofimage scaling. For an interpolation factor of two, each pixel in thelow-resolution image is the average of the corresponding 2×2 block ofpixels in the monochrome image. Specifically, the pixel LR(i,j) in thei^(th) row and j^(th) column of the low-resolution image is computedaccording to equation (9): $\begin{matrix}{{L\quad {R\left( {i,j} \right)}} = \frac{{T\left( {{2i},{2j}} \right)} + {T\left( {{{2i} + 1},{2j}} \right)} + {T\left( {{2i},{{2j} + 1}} \right)} + {T\left( {{{2i} + 1},{{2j} + 1}} \right)}}{4}} & (9)\end{matrix}$

[0052] where T(i,j) is the i^(th) row and j^(th) column of the trainingimage. If the monochrome training image has a height of H pixels and awidth of W pixels, the low-resolution image has a height of H/2 pixelsand a width of L/2 pixels, the number of pixels being rounded down to aninteger.

[0053] A sharpened high-resolution rendering of the monochrome image mayalso be computed (step 104). The high-resolution image can be sharpenedusing an unsharp mask. A pixel SHR(i,j) in a sharpened, high-resolutionimage can be computed as follows: $\begin{matrix}{{A\left( {i,j} \right)} = {\frac{1}{9}{\sum\limits_{m = {- 1}}^{1}{\sum\limits_{n = {- 1}}^{1}{T\left( {{i + m},{j + n}} \right)}}}}} & (10)\end{matrix}$

 SHR(i,j)=T(i,j)+λ(T(i,j)−A(i,j))  (11)

[0054] where λ is 1.0. Generally, λ is a non-negative constant whichcontrols the level of sharpening and A(i,j) is an average of the pixelsin a neighborhood around pixel (i,j). The dimensions of the sharpened,high-resolution image are the same as the dimensions of the monochromeimage. This step 104 is optional. The uncorrupted training image can beused in place of the sharpened image. However, sharpening the trainingimage improves edge quality in the interpolated image without the needfor post-processing. For exemplary purposes, the training method will bedescribed in connection with the sharpened high-resolution image.

[0055] A number NCV of cluster vectors are then extracted from thelow-resolution image (block 106). The cluster vectors can be extractedfrom an observation window of any size and shape. Although the clustervector extraction will be described hereinafter in connection with a 3×3pixel observation window, a larger window such as a 5×5 pixel windowcould be used.

[0056] As for the number NCV of cluster vectors that may be extractedfrom the low resolution image, there are as many cluster vectorsavailable as there are low-resolution pixels that are not on an imageborder. A reasonable range of numbers of cluster vectors is between25,000 and 100,000. Using 25,000 cluster vectors will yield afair-quality classifier at moderate computational cost, while using100,000 cluster vectors will yield a high-quality classifier atsignificantly greater computational cost.

[0057] The cluster vectors are extracted from spatially differentregions of the low-resolution image. To ensure this, the pixels aresampled at a period S1=N/NCV, where N is the total number of clustervectors available. The sampling period S1 is truncated to an integer.Thus, a cluster vector is extracted at every S1^(th) pixel of thelow-resolution image. The pixels are sampled in a raster ordering, withthe primary direction being from left to right and with the secondarydirection being from top to bottom (the “mastering order”).

[0058] For a 3×3 observation window, the cluster vector can be computedaccording to equations (2) to (5). However, it is not required tosubtract the nominal value from the neighboring pixels in the firstvector y1, nor is it required to scale the second vector y2. However,subtracting the nominal value and scaling will make it easier to discernthe different types of cluster vectors. The purpose and effect ofscaling is to warp the space of all of the cluster vectors so that edgesof different magnitude but similar orientation or shape are groupedtogether.

[0059] After the cluster vectors have been extracted, a number M isselected (block 108). The number M represents the number ofrepresentative vectors that will be computed for the low-resolutionimage. The number M might be between 50 and 100. Larger numbers M ofrepresentative vectors will allow for more freedom because a widervariety of behaviors can be represented and used for resolutionsynthesis. However, the larger numbers M of representative vectors willalso require more memory and computational resources.

[0060] After the number M is selected, the classifier (i.e., therepresentative vectors RV, the class weights CW and the variance VAR)are computed (blocks 110 to 118). A method of maximum likelihoodestimation can be used to compute the classifier. One instantiation ofthe method of maximum likelihood estimation is the well-knownexpectation-maximization (EM) algorithm applied to Gaussian mixturemodels. Initial values for the classifier are selected and theniteratively updated until a locally optimal set of parameters has beenobtained.

[0061] Thus, initial values are selected for the classifier (block 110).The initial values for the class weights CW can be selected as follows:$\begin{matrix}{{{C\quad W_{0}^{(0)}} = \frac{1}{M}},\ldots \quad,{{C\quad W_{M - 1}^{(0)}} = \frac{1}{M}}} & (12)\end{matrix}$

[0062] The superscripts “(0)” indicate initial (zeroth) values for theclass weights CW. The class weights CW are positive numbers that addup 1. That is, CW₀>0, . . . , CW_(M−1)>0 and ΣCW_(i)=1 from i=0 toi=M−1.

[0063] The initial values for the representative vectors RV₀, . . . ,RV_(M−1) can be set equal to the cluster vectors sampled at everyS2^(th) pixel of the low resolution image, where the sampling periodS2=NCV/M. Thus,

RV_(i) ⁽⁰⁾=y_(S2xi)  (13)

[0064] for i=0, . . . , M−1. The sampling period S2 ensures that therepresentative vectors RV₀, . . . , RV_(M−1) are initialized by clustervectors from spatially different regions of the low-resolution image.The cluster vectors are selected from spatially separate regions of thelow resolution image (or from different images if multiple images areavailable for training) to encourage the formation of distinctrepresentative vectors.

[0065] The variance VAR can be initialized as follows: $\begin{matrix}{{VAR}^{(0)} = {\frac{1}{{NCV} \times 8}{\sum\limits_{m = 0}^{7}{\sum\limits_{i = 0}^{{NCV} - 1}\left( {{y_{i}(m)} - {\overset{\_}{y}(m)}} \right)^{2}}}}} & (14)\end{matrix}$

[0066] where y_(i)(m) is the m^(th) element in the i^(th) clustervector, and {overscore (y)}(m) is the sample mean of the m^(th) elementin the cluster vectors. The sample mean {overscore (y)}(m) can becalculated as follows. $\begin{matrix}{{\overset{\_}{y}(m)} = {\frac{1}{NCV}{\sum\limits_{i = 0}^{{NCV} - 1}{y_{i}(m)}}}} & (15)\end{matrix}$

[0067] Next a log likelihood LL(kl) is computed for the initial values(block 112). A log likelihood LL(yi;kl) for each cluster vector iscalculated according to equations (16) and (17), and the log likelihoodLL(kl) is computed by adding the log likelihoods of each cluster vectorsaccording to equation (18): $\begin{matrix}{{{y_{i} - {R\quad V_{j}^{({k\quad l})}}}}^{2} = {\sum\limits_{m = 0}^{7}\left( {{y_{i}(m)} - {R\quad {V_{j}^{({k\quad l})}(m)}}} \right)^{2}}} & (16) \\{{L\quad {L\left( {y_{i};{k\quad l}} \right)}} = {\log \left\lbrack {\sum\limits_{j = 0}^{M - 1}{\frac{C\quad W_{j}^{k\quad l}}{\left( {2\pi \quad {VAR}^{({k\quad l})}} \right)^{4}}{\exp \left( {\frac{- 1}{2{VAR}^{({k\quad l})}}{{y_{i} - {R\quad V_{j}^{({k\quad l})}}}}^{2}} \right)}}} \right\rbrack}} & (17) \\{{L\quad {L\left( {k\quad l} \right)}} = {\sum\limits_{i = 0}^{{NCV} - 1}{L\quad {L\left( {y_{i};{k\quad l}} \right)}}}} & (18)\end{matrix}$

[0068] where the index kl indicates the kl^(th) iteration (for example,LL(0) is the log likelihood for the initial values); y_(i)(m) is them^(th) element in the i^(th) cluster vector; RV_(j) ^((kl))(m) is them^(th) element of the j^(th) representative vector in the kl^(th)iteration; LL(y_(i); kl) is the log likelihood of an individual clustervector y_(i) during the kl^(th) iteration; log is base e, and “exp”denotes the exponential function. The log likelihood is a number that isanalogous to a measure of the quality of the current values for theclassifier. A higher log likelihood indicates a better “fit” for theclassifier.

[0069] After the log likelihood LL(0) is computed for the initial valuesof the classifier, the classifier is updated. The index kl isincremented (kl→kl+1) and the updates are made as follows.$\begin{matrix}{{p\left( {\left. j \middle| y_{i} \right.;{{k\quad l} - 1}} \right)} = \frac{{\exp \left( \frac{- {{y_{i} - {R\quad V_{j}^{({{k\quad l} - 1})}}}}^{2}}{2V\quad A\quad R^{({{k\quad l} - 1})}} \right)}C\quad W_{j}^{({{k\quad l} - 1})}}{\sum\limits_{d = 0}^{M - 1}{{\exp \left( \frac{- {{y_{i} - {R\quad V_{d}^{({{k\quad l} - 1})}}}}^{2}}{2V\quad A\quad R^{({{k\quad l} - 1})}} \right)}C\quad W_{d}^{({{k\quad l} - 1})}}}} & (19) \\{{N\quad C\quad V_{j}^{({k\quad l})}} = {\sum\limits_{i = 0}^{{N\quad C\quad V} - 1}{p\left( {\left. j \middle| y_{i} \right.;{{k\quad l} - 1}} \right)}}} & (20) \\{{C\quad W_{j}^{({k\quad l})}} = \frac{N\quad C\quad V_{j}^{({k\quad l})}}{N\quad C\quad V}} & (21) \\{{R\quad V_{j}^{({k\quad l})}} = {\frac{1}{N\quad C\quad V_{j}^{({k\quad l})}}{\sum\limits_{i = 0}^{{N\quad C\quad V} - 1}{y_{i}{p\left( {\left. j \middle| y_{i} \right.;{{k\quad l} - 1}} \right)}}}}} & (22) \\{{V\quad A\quad R^{({k\quad l})}} = {\frac{1}{8}{\sum\limits_{j = 0}^{M - 1}{\frac{C\quad W_{j}^{({k\quad l})}}{N\quad C\quad V_{j}^{({k\quad l})}}{\sum\limits_{i = 0}^{{N\quad C\quad V} - 1}{{{y_{i} - {R\quad V_{j}^{({k\quad l})}}}}^{2}{p\left( {\left. j \middle| y_{i} \right.;{{k\quad l} - 1}} \right)}}}}}}} & (23)\end{matrix}$

[0070] The parameter p(j|y_(i);kl−1) can be considered a measure of theextent to which a cluster vector y_(i) belongs to the jth representativevector of the (kl−1)^(th) iteration. NCV(kl) can be considered as anestimate of the number of cluster vectors in the jth class of theprevious iteration. The updates in equations (21), (22) and (23) can beconsidered as sample statistics that are weighted and normalized inaccordance with the classifications under the previous iteration.

[0071] Next, a log likelihood LL(1) is computed for the updatedclassifier (block 116). The log likelihood LL(1) for the updatedclassifier is also computed according to the equations (16), (17) and(18) above.

[0072] A decision is then made as to whether the current values for therepresentative vectors RV, the class weights CW and the variance VAR areoptimal (block 118). The decision can be made by taking the differencebetween the log likelihoods of the current and previous iterations LL(1)and LL(0) and comparing the difference to a threshold THRES. Thethreshold THRES indicates whether the likelihood is approaching a localmaximum. The threshold THRES can be computed as follows:

THRES=0.09×log(8×NCV)  (24)

[0073] where the log is base e. If the difference is greater than thethreshold THRES, another update is performed (block 114). If thedifference is less than the threshold THRES, the values of the kl^(th)iteration for the classifier are stored in the database (block 120).

[0074] After the classifier has been computed, the coefficients a, b forthe interpolation filters are computed (blocks 122 to 126). A numberNFDV of filter design triplets are computed from pixels in the lowresolution image (block 122). Each filter design vector triplet includesa cluster vector y, an observation vector L which containslow-resolution pixels, and a vector h which contains high-resolutionpixels. All three of these vectors y, L and h correspond to exactly onepixel in the low-resolution image. The set of filter design vectortriplets will be referred to as${\left\{ \left( {y_{i},L_{i},h_{i}} \right) \right\} \begin{matrix}{{N\quad F\quad D\quad V} - 1} \\{i = 0}\end{matrix}},$

[0075] where (y_(i),L_(i),h_(i)) is the filter design vector triplet forthe i^(th) pixel in the low-resolution image.

[0076] As for the number NFDV of design triplets, there may be as manyfilter design vector triplets available as there are low-resolutionimage pixels from which cluster vectors may be extracted. Higher numbersNFDV of filter design vector triplets will yield better results at thecost of increased computation. Satisfactory results can be obtained byusing 500,000 filter design vector triplets, while better results can beobtained by using over 1,000,000 filter design vector triplets.

[0077] The high-resolution pixel vector h is computed by stacking a 2×2block of high-resolution pixels which correspond to a pixel in thelow-resolution image. Specifically for a low-resolution pixel LR(i,j) atlocation i,j in the low resolution image, the corresponding pixels inthe 2×2 block of the high resolution image are at locations (2*i,2*j),(2*i+1, 2*j), (2*i, 2*j+1), and (2*i+1, 2*j+1).

[0078] The filter design triplets can be sampled from spatiallydifferent regions of the low-resolution image. A sampling period S3might be N/NFDV, where N is the total number of filter design vectortriplets available. The sample period S3 is truncated to an integer. Oneby one, in the rastering order, a filter design vector triplet isextracted at every S3^(th) pixel in the low-resolution image.

[0079] Next, training statistics are computed for each representativevector (block 124). Training statistics for the j^(th) representativevector can be computed as follows: $\begin{matrix}{{N\quad F\quad D\quad V_{j}} = {\sum\limits_{i = 0}^{{N\quad F\quad D\quad V} - 1}{p\left( j \middle| y_{i} \right)}}} & (25) \\{{m\quad L_{j}} = {\frac{1}{N\quad F\quad D\quad V_{j}}{\sum\limits_{i = 0}^{{N\quad F\quad D\quad V} - 1}{L_{j}{p\left( j \middle| y_{i} \right)}}}}} & (26) \\{{m\quad h_{j}} = {\frac{1}{N\quad F\quad D\quad V_{j}}{\sum\limits_{i = 0}^{{N\quad F\quad D\quad V} - 1}{h_{i}{p\left( j \middle| y_{i} \right)}}}}} & (27) \\{{G\quad L\quad L_{j}} = {\frac{1}{N\quad F\quad D\quad V_{j}}{\sum\limits_{i = 0}^{{N\quad F\quad D\quad V} - 1}{\left( {L_{i} - {m\quad L_{j}}} \right)\left( {L_{i} - {m\quad L_{j}}} \right)^{t}{p\left( j \middle| y_{i} \right)}}}}} & (28) \\{{G\quad h\quad L_{j}} = {\frac{1}{N\quad F\quad D\quad V_{j}}{\sum\limits_{i = 0}^{{N\quad F\quad D\quad V} - 1}{\left( {h_{i} - {m\quad h_{j}}} \right)\left( {L_{i} - {m\quad L_{j}}} \right)^{t}{p\left( j \middle| y_{i} \right)}}}}} & (29)\end{matrix}$

[0080] where mL_(j) is a 9-dimensional vector and mh_(j) is a4-dimensional vector, GLL_(j) is a 9×9 matrix, and GhL_(j) is a 4×9matrix. Superscript “t” denotes the vector transpose.

[0081] Finally, the coefficients a, b for the interpolation filters arecomputed from the training statistics (block 126). Each interpolationfilter produces a pixel in the high resolution image. The set ofinterpolation filters can be represented as${\left\{ {\left\{ {{a\left( {j,k} \right)},{b\left( {j,k} \right)}} \right\} \begin{matrix}3 \\{k = 0}\end{matrix}} \right\} \begin{matrix}{M - 1} \\{j = 0}\end{matrix}},$

[0082] where k represents the k^(th) output pixel in the high resolutionimage (for k=0, . . . 3). The vector a(j,k) is the kth row of a 4×9matrix Aj, and the scalar b(j,k) is the k^(th) element in a4-dimensional vector bj. The matrix Aj and the vector bj can be computedaccording to equations (30) and (31). The superscript “−1” denotestaking the matrix inverse (or pseudo-inverse, if necessary).

A _(j) =GhL _(j)(GLL _(j))⁻¹  (30)

b _(j) =mh _(j) −GhL _(j)(GLL _(j))⁻¹ mL _(j)  (31)

[0083] The coefficients for the interpolation filters are stored in thedatabase 18 (block 128). The database 18 can be stored on a medium suchas CD-ROM for transfer to another computer, it can be transferred toanother computer or printer via a network, or it can be programmed intoan EEPROM of a printer (block 130). The database 18 can be used asdescribed above in connection with FIGS. 2a to 2 c.

[0084] Thus disclosed are an interpolation method for producing highquality interpolated images and a training method for generating adatabase of parameters for the interpolation method. The training methodis run off-line in a computer. The database of parameters need only begenerated once, but it may be used many times by the interpolationmethod. Once the parameters have been generated, the interpolationmethod is easy to implement in terms of processing power and memoryrequirements.

[0085] Although the system shown in FIG. 1 was described in connectionwith a printer, the system is not so limited. The invention could beapplied to the Internet. For example, many web pages display thumbnailsof images. Clicking on a thumbnail causes a higher resolution image tobe downloaded. If a web browser is capable of performing resolutionsynthesis according to the present invention, the higher resolutionimage need not be downloaded when the thumbnail is clicked. Instead, theweb browser would access the database (from either its hard drive or theserver) and generate a high-resolution image from the thumbnail. Thehigh-resolution image would be displayed much faster than downloading ahigh resolution image file over a modem connection. Additionally, lessserver memory would be needed to store the web pages.

[0086] The invention can be used by any computer to increase resolutionof a displayed image. For example, the interpolation method can be usedby a computer to zoom in on an area of an image. The zoomed-in areawould have smooth, continuous edges.

[0087] Moreover, the invention is not limited to image processing. Theinvention could be applied to the enhancement or restoration of speech,still image, video, and other multidimensional signals.

[0088]FIG. 7 shows a more generalized method of building a database thatcould be applied to resolution synthesis of speech or the restoration ofimages. Resolution synthesis parameters are generated from at least onetraining file. All training files are inputted (block 300),low-resolution files are generated from the training files (block 302),and a plurality of representative vectors are computed from thelow-resolution files (block 304). A set of interpolation filters isgenerated for each of the representative vectors (block 306). Theinterpolation filters and the representative vectors are stored in thedatabase (block 308) and can be used later to perform data resolutionsynthesis on a file other than the training file. If generated fromtraining files including speech pattern data, the database can be usedby a computer to restore or synthesize low-quality audio filescontaining speech. Similarly, if generated from training files includingdata representing a damaged image, the database can be used by acomputer to restore the image.

[0089] Specific embodiments of the invention have been described andillustrated above. However, the invention is not limited to thesespecific embodiments. Any number and type of training images can be usedto create the database. The low-resolution images can be computed in anynumber of ways other than block averaging. The optional sharpened imagescan be computed in any number of ways other than unsharp masking.

[0090] The actual number NCV of cluster vectors will depend upon thecomputational power of the computer that is generating the database andthe desired quality of the interpolated images. Increasing the numberNCV of the cluster vectors will also increase the computational time.However, increasing the number NCV of cluster vectors will increase theamount of information used for interpolation and, therefore, willincrease the quality of the interpolated image. Therefore, it might befeasible to use far more than 100,000 cluster vectors.

[0091] The same holds true of the number M of classes. Increasing thenumber M of classes will increase computation time. However, increasingthe number of classes will increase the variety of behaviors that can berecognized and used to perform image interpolation. A single class couldeven be used, in which case the best class would be selected. If only asingle class is selected, class weights would not have to be computed.

[0092] It is assumed that the observation window gives all theinformation needed to interpolate a sampled pixel. Of course, theassumption will hold truer for larger observation windows. Therefore, itmight be desirable to use an observation window that is larger than 3×3pixels.

[0093] Generating the representative vectors is not limited to theexpectation maximization technique. Any of a large number of suitableparameter optimization techniques could be used for generatingrepresentative vectors from a group of cluster vectors. Examples includeconjugate gradient and gradient search techniques and simulatingannealing.

[0094] Generating the cluster vectors is not limited to the methodsdescribed above. The cluster vectors could be generated by a widevariety of clustering methods such as iso-data clustering, K-meansclustering, general vector quantization, and tree-structurequantization.

[0095] Therefore, the invention is not limited to the specificembodiments described and illustrated above. Instead, the invention isconstrued according to the claims that follow.

What is claimed is:
 1. A method of building an electronic database fordata resolution synthesis from at least one training file, the methodcomprising the steps of: generating a low-resolution file from eachtraining file; generating a plurality of representative vectors fromeach low-resolution file; and generating a set of interpolation filtersfor each of the representative vectors; whereby the interpolationfilters and the representative vectors can be used to perform dataresolution synthesis on a file other than the training file.
 2. Themethod of claim 1, wherein the representative vectors are generated bycomputing a number NCV of cluster vectors from each low-resolution fileand using the cluster vectors to compute the representative vectors; andwherein low-resolution observation vectors, the cluster vectors, therepresentative vectors and a high-resolution file corresponding to eachlow-resolution file are used to compute the interpolation filters,whereby a high resolution file may be a training file.
 3. The method ofclaim 2, further comprising the step of generating a sharpenedhigh-resolution file, the sharpened high-resolution file being used tocompute the interpolation filters.
 4. The method of claim 2, wherein therepresentative vectors are generated by using a maximum likelihoodestimate.
 5. The method of claim 4, wherein the vectors are generated byusing an expectation maximization technique.
 6. The method of claim 4,wherein a classifier including the representative vectors is computed byinitializing the classifier and updating the classifier until optimalvalues for the classifier have been obtained.
 7. The method of claim 6,wherein the classifier further includes a variance and a number M ofclass weights, and wherein the representative vectors, the class weightsand the variance are computed simultaneously.
 8. The method of claim 2,wherein each cluster vector is generated by forming an observationwindow about sampled data in a low resolution file, extracting a vectorincluding neighboring data of the sampled data, and scaling the vector.9. The method of claim 2, wherein coefficients for the interpolationfilters are computed by: computing a number NFDV of filter designtriplets from data in the low-resolution file, where NFDV is a positiveinteger, each filter design triplet corresponding to sampled data in thelow-resolution file, each filter design triplet including an observationvector for the sampled data, a cluster vector for the sampled data, anda vector of high resolution data from a high-resolution file; computingtraining statistics from the filter design triplets; and computing thecoefficients from the training statistics.
 10. The method of claim 2,wherein the steps are run off-line in a computer.
 11. The method ofclaim 1, wherein the interpolation filters are linear filters.
 12. Themethod of claim 1, wherein the representative vectors are generated byusing a parameter optimization technique.
 13. A method of using acomputer to compute a plurality of resolution synthesis parameters froma training image, the method comprising the steps of: computing alow-resolution image from the training image; computing a plurality ofcluster vectors for a number NCV of pixels in the low-resolution image,where NCV is a positive integer; using the cluster vectors to compute anumber M of representative vectors for the low resolution image, where Mis a positive integer that is less than NCV; and using low-resolutionobservation vectors, the cluster vectors, the representative vectors andvectors from a high-resolution image to compute sets of interpolationfilter coefficients corresponding to each of the representative vectors;whereby the high-resolution image may be the training image; and wherebythe interpolation filter coefficients and the number M of representativevectors are stored in the database for later interpolation of an imageother than the training image.
 14. The method of claim 13, wherein thenumber NCV is between 25,000 and 100,000, whereby between 25,000 and100,000 cluster vectors are computed.
 15. The method of claim 13,wherein each cluster vector for a non-border pixel is computed byextracting a first vector from a square observation window centeredabout a sampled pixel in the low-resolution image, and scaling the firstvector.
 16. The method of claim 13, where the number M of representativevectors is between 50 and
 100. 17. The method of claim 13, wherein therepresentative vectors are computed using a maximum likelihood estimate.18. The method of claim 17, wherein a classifier including therepresentative vectors is computed by initializing the classifier andupdating the classifier until optimal values for the classifier havebeen obtained.
 19. The method of claim 13, wherein the representativevectors are computed using an expectation-maximization algorithm. 20.The method of claim 19, wherein the representative vectors are computedby: setting initial values for a classifier including a number M ofclass weights, a variance and the number M of representative vectors;computing a quality measure of how well the cluster vectors arerepresented by the initial values for the classifier; updating theclassifier; recomputing the quality measure for the updated classifier;and determining whether the cluster vectors are suitably represented bythe updated classifier, the classifier being updated until the clustervectors are suitably represented.
 21. The method of claim 13, furthercomprising the step of computing a sharpened high-resolution image fromthe training image, wherein the sharpened image is used along withlow-resolution observation vectors, the cluster vectors and therepresentative vectors to compute the interpolation filter coefficients.22. The method of claim 13, wherein the interpolation filtercoefficients are computed by: computing a number NFDV of filter designtriplets from the low-resolution image, where NFDV is a positiveinteger, each filter design triplet corresponding to a sampled pixel inthe low-resolution image, each filter design triplet including anobservation vector for the sampled pixel, a cluster vector for thesampled pixel, and a vector of high-resolution pixels corresponding tothe sampled pixel, the high-resolution pixels being taken from thehigh-resolution image; computing training statistics from the filterdesign triplets; and computing the coefficients from the trainingstatistics.
 23. The method of claim 22, wherein the number NFVD offilter design triplets is between 500,000 and 1,000,000, whereby between500,000 and 1,000,000 filter design triplets are computed.
 24. Themethod of claim 22, wherein the interpolation filter coefficients arecomputed for linear interpolation filters.
 25. The method of claim 13,wherein the steps are run off-line in the computer.
 26. The method ofclaim 25, wherein the database is stored for transfer to a secondcomputer, whereby the second computer can access the database to performimage interpolation on images other than the training images.
 27. Themethod of claim 25, wherein the database is stored in memory of aprinter, whereby the printer can access the database to perform imageinterpolation on images other than the training images.
 28. The methodof claim 13, wherein the representative vectors are generated by using aparameter optimization technique.
 29. Apparatus comprising: a processor;and memory means for storing an electronic database and a plurality ofexecutable instructions, the instructions, when executed, instructingthe processor to access a training file; generate a low-resolution filefrom the training file; generate a plurality of representative vectorsfrom the low-resolution file; generate a set of interpolation filtersfor each of the representative vectors; and store the interpolationfilters and the representative vectors in the memory means as part ofthe database.
 30. The apparatus of claim 29, wherein the instructionsinstruct the processor to generate the representative vectors bycomputing a number NCV of cluster vectors from the low-resolution file,and using the cluster vectors to generate the representative vectors;and wherein the instructions instruct the processor to generate theinterpolation filters from low-resolution observation vectors, thecluster vectors, the representative vectors and a plurality of vectorsfrom a high-resolution file corresponding to the low-resolution file.31. The apparatus of claim 30, wherein the instructions further instructthe processor to generate a sharpened high-resolution file from thetraining file, the sharpened high-resolution file being used to computethe interpolation filters.
 32. The apparatus of claim 30, wherein theinstructions instruct the processor to generate a classifier includingthe representative vectors by initializing the classifier and updatingthe classifier until optimal values for the classifier have beenobtained.
 33. The apparatus of claim 30, wherein the instructionsinstruct the processor to generate each cluster vector by forming anobservation window about sampled data in the low-resolution file,extracting a vector including neighboring data of the sampled data,subtracting a value of the sampled data from values of the data in thevector; and scaling the vector.
 34. The apparatus of claim 30, whereinthe instructions instruct the processor to compute coefficients for theinterpolation filters by: computing a number NFDV of filter designtriplets from data in the low-resolution file, where NFDV is a positiveinteger, each filter design triplet corresponding to sampled data in thelow-resolution file, each filter design triplet including an observationvector for the sampled data, a cluster vector for the sampled data, anda vector of high resolution data from a high-resolution file, the highresolution data corresponding to the sampled data; computing trainingstatistics from the filter design triplets; and computing thecoefficients from the training statistics.
 35. The apparatus of claim30, wherein the interpolation filters are linear filters.
 36. An articleof manufacture for instructing a processor to compute a resolutionsynthesis database from a training image, the article comprising:computer memory; and a plurality of executable instructions stored inthe computer memory, the instructions, when executed, instructing theprocessor to compute a low-resolution image from the training image;compute a plurality of representative vectors from the low-resolutionimage; and compute a set of interpolation filters for each of therepresentative vectors; whereby the interpolation filters and therepresentative vectors form a part of the database.
 37. The article ofclaim 36, wherein the instructions instruct the processor to compute therepresentative vectors by computing a number NCV of cluster vectors fromthe low-resolution image, and using the cluster vectors to compute therepresentative vectors; and wherein the instructions instruct theprocessor to compute the interpolation filters from low-resolutionobservation vectors, the cluster vectors, the representative vectors andvectors from a high-resolution image corresponding to the low-resolutionimage.
 38. The article of claim 37, wherein the instructions furtherinstruct the processor to compute a sharpened high-resolution image fromthe training image, the sharpened high-resolution file being used tocompute the interpolation filters.
 39. The article of claim 37, whereinthe instructions instruct the processor to compute a classifierincluding the representative vectors by initializing the classifier andupdating the classifier until optimal values for the classifier havebeen obtained.
 40. The article of claim 37, wherein the instructionsinstruct the processor to compute each cluster vector by forming anobservation window about a sampled pixel in the low-resolution image,extracting a vector including neighboring pixels of the sampled pixel,subtracting a value of the sampled pixel from values of the pixels inthe vector; and scaling the vector.
 41. The article of claim 37, whereinthe instructions instruct the processor to compute coefficients for theinterpolation filters by: computing a number NFDV of filter designtriplets from pixels in the low-resolution image, where NFDV is apositive integer, each filter design triplet corresponding to a sampledpixel in the low-resolution image, each filter design triplet includingan observation vector for the sampled pixel, a cluster vector for thesampled pixel, and a vector of high resolution pixels from ahigh-resolution image, the high resolution pixels corresponding to thesampled pixel; computing training statistics from the filter designtriplets; and computing the coefficients from the training statistics.42. The article of claim 36, wherein the representative vectors aregenerated by using a parameter optimization technique.
 43. An article ofmanufacture comprising: computer memory; and a database encoded in thecomputer memory, the database including a plurality of sets ofresolution synthesis parameters, each set corresponding to aninterpolation factor, each set including a classifier and a number M ofresolution synthesis filters, each classifier including a number M ofrepresentative vectors, where M is a positive integer.
 44. The articleof claim 43, wherein each classifier further includes a variance and anumber M of class weights.
 45. The article of claim 43, wherein thenumber M is between 50 and 100.